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Method for Efficient Computation of Image Frames for Dual- 
Modulation Display Systems Using Key Frames 

Real time Video Computation for High Dynamic Range Displays 

A dual modulation high dynamic range (HDR) display (see US patent application serial number 
10/469,473, incorporated herein by reference) uses two image modulators in series. Three key 
implementations have been designed so far: A controlled array of light emitting diodes (LED) or 
other light sources projecting light through a liquid crystal display (LCD), a bright backlight 
projecting light through two LCDs in series, and a video projector projecting light through an LCD. 
In order to avoid alignment artifacts between the two modulators each design uses a varying 
degree of resolution mismatch between the two modulators where the front LCD is high resolution 
colour and the rear modulators (LED, first LCD, projector respectively) is low resolution. The front 
image is modified to correct for the blur of the rear image such that the combined output image is 
a high resolution HDR image. In the following we describe the first (LED based) design for the 
discussion but the method applies to all HDR display designs with a resolution mismatch between 
the modulators. 

One of the drawbacks of this system is that the rendering (computation on the graphic card) of 
the image couplet is very computationally expensive. In principle, images are rendering in the 
following way: 

1 . A floating point or higher than 8-bit image ("input image") is provided to the algorithm 

2. Appropriate intensities for each LED are established. The corresponding driving levels 
are stored ("LED Values"). 

3 The effective luminance distribution from the entire LED array is calculated by adding the 
modulated point spread functions (PSF) of the LEDs together. This is very costly as each 
image pixel has to be rendered multiple times to account for light coming from all LEDs 
with a PSF that stretches onto that pixel. 

4. The input image is divided by the effective luminance distribution to yield the LCD image. 

5. Both LCD image and LED values are sent to the display 

6. At the display the LED array is modulated by the incoming LED values to create the 
effective luminance distribution which is then optically modulated by the transmission of 
the LCD according to the LCD image. The final image output corresponds to the input 
image. 

7. This process is repeated for the next image frame. 

This invention describes a method to reduce the computation time associated with the third step, 
which is the most costly. Instead of following the above procedure for each frame of the image we 
can assume that from one frame to the next there are only small changes to the image. As a 
result it is possible to calculate a single LED image (LED values and the associated effective 
luminance distribution) and then use it for one or more additional frames. The LCD image will be 
continually modified to make the necessary changes from frame to frame but this is 
computationally much less costly. By using an LED key frame every third frame the computation 
cost would be approximately 1/3 of the cost associated with frame by frame computation and the 
procedure would look like this: 

1. calculate LED key frame 

2. divide first frame by LED key frame to get first LCD image 

3. display LED key frame and first LCD image 

4. divide second frame by LCD key frame to get second LCD image 

5. display LED key frame and second LCD image 

6. divide third frame by LCD key frame to get third LCD image 
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7. display LED key frame and third LCD image 

8. calculate second LED key frame 

9. divide fourth frame by second LED key frame to get fourth LCD image 

10. display second LED key frame and fourth LCD image 

11. etc 

As will be obvious to one skilled in the art, very simple modifications to this procedure will enable 
the use of LED key frame changes every second, fourth, fifth ... nth, frame. 

The frames between key frames will be very accurate as long as the luminance changes during a 
key frame sequence are minor and can be contained within the dynamic range of the LCD alone 
(200-1 for a standard LCD which is significantly higher that most frame-to-frame luminance 
changes). The only artifacts introduced by this method occur if in a particular key frame the LCD 
dynamic range is already used to the maximum to correct for a high contrast boundary in the 
image and that boundary becomes even higher contrast in a subsequent non-key frame. These 
situations are very rare and are unlikely to be visible at video rates. 

These artifacts can be further reduced by a dynamic use of the key frames. Instead of fixing their 
occurrence to once ever few frames it is possible to establish one key frame and then keep count 
of the number of pixel per new frame which are not accurately reproducibly just by modifying the 
LCD Image. If a certain threshold is exceeded the software renders a new key frame and 
otherwise no changes are necessary. Since the process of dividing the new frame by the LED 
key frame is very fast (linear process of one division operation per pixel) this could happen 
without any major delay. If the number of artifact pixels is higher than the threshold then the 
software calculates and uses a new LED key frame. If not, then the calculations for this frame are 
done (the division has already yielded the LCD image in addition to the number of artifact pixel). 
Such a dynamic key frame structure could dramatically reduce computation time for some 
applications where for long periods no major luminance changes occur in a scene. 

If a key frame needs to be computed then the resulting lag can be distributed over several 
frames. In the interim period between the failure of the old key frame and the finished calculation 
of the new key frame it is possible to use different approximation of the accurate key frame. Any 
choice of combination from the list of options is appropriate: 

• The interim frames can use a standard key frame. A standard key frame would be an 
approximately appropriate pattern that has been pre-computed. The system can rapidly 
execute a search through a list of pre-computed pattern to determine which is most suitable 
for the interim period. Examples of common pre-computed key frames are 

- all LEDs on at mid level 

- all LEDs full on 

- half or quarter of the LEDs full on 

- half or quarter of the LEDs half on 
all previously computed key frames 

. It is also possible to compute an interim key frame rapidly by relaxing the accuracy 
requirements of the LED point spread simulation (i.e. simulating the actual light distribution 
per LED with an approximate Gaussian, etc). This method will reduce the lag between key 
frame computations and the introduced imperfections are unlikely to be visible to the viewer 
as the result will be approximately accurate and perfect accuracy will be reached very rapidly 
once the new keyframe is available. The accuracy of this approximation can be increased at 
each frame until the next perfect keyframe has been established 

• Since the pixels that cannot be reproduced with the old key frame are known it is possible to 
replace just the LED values corresponding to those pixels. Since the LED values of the old 
keyframe are known it is possible to locate the inappropriate LED, subtract its point spread 
function multiplied by the corresponding LED value in the old key frame from the old key 
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frame. This effectively provides the LED luminance pattern under the old key frame setting 
with the inappropriate LED missing. The next step is to establish the appropriate LED value 
by the usual method. This new value multiplied by the point spread function of the LED can 
then be added to the modified key frame. This process is repeated as often as necessary 
until all inappropriate LED values have been replaced. The key frame is now appropriate for 
the image again. This update can take several frames and should proceed in the order of 
greatest error (i.e. the first LED to be replaced is the one with the largest number of pixels 
that cannot achieve the required value with the old key frame.) 
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